Objectives

This notebook will demonstrate how to:

  • Identify when Gene Set Variation Analysis (GSVA) is well-suited for an analysis
  • Perform GSVA on transformed RNA-seq data with the GSVA package
  • Explore the dependence of GSVA scores on gene set size with random gene sets

So far every pathway analysis method we’ve covered relies on some information about groups of samples in our data. For over-representation analysis (ORA), we created gene sets from two different two group comparisons. In the Gene Set Enrichment Analysis (GSEA) example, we used statistics from a differential gene expression (DGE) analysis where we compared MYCN amplified cell lines to non-amplified cell lines; we needed that amplification status information.

What if we’re less sure about groups in our data or we want to analyze our data in a more unsupervised manner?

In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) (Hänzelmann et al. 2013) that allows us to calculate gene set or pathway scores on a per-sample basis.

We like this quote from the GSVA paper (Hänzelmann et al. 2013) to set the stage:

While [gene set enrichment] methods are generally regarded as end points of a bioinformatic analysis, GSVA constitutes a starting point to build pathway-centric models of biology.

Rather than contextualizing some results you already have from another analysis like DGE, GSVA is designed to provide an estimate of pathway variation for each of the samples in an experiment. Note that these scores will depend on the samples included in the dataset when you run GSVA; if you added more samples and reran GSVA, you would expect the scores to change.

Set up

Libraries

# Pipes
library(magrittr)
# Gene Set Variation Analysis
library(GSVA)

Directories and files

Directories

# We have some medulloblastoma data from the OpenPBTA project that we've 
# prepared ahead of time
input_dir <- file.path("data", "open-pbta")

# Create a directory specifically for the results using this dataset
output_dir <- file.path("results", "open-pbta") 
if (!dir.exists(output_dir)) {
  dir.create(output_dir, recursive = TRUE)
}

Input

We have VST transformed RNA-seq data, annotated with gene symbols, that has been collapsed such that there are no duplicated gene identifiers (see setup).

rnaseq_file <- file.path(input_dir, "medulloblastoma_vst_collapsed.tsv")

Output

gsva_results_file <- file.path(output_dir, "medulloblastoma_gsva_results.tsv")

Gene sets

The function that we will use to run GSVA wants the gene sets to be in a list, rather than a tidy data frame that we used with clusterProfiler (although it does accept other formats).

We’re going to take this opportunity to introduce a different format that gene sets are often distributed in called GMT (Gene Matrix Transposed).

We’re going to read in the Hallmark collection file directly from MSigDB, rather than using msigdbr like we did in earlier notebooks.

The RNA-seq data uses gene symbols, so we need gene sets that use gene symbols, too.

# R can often read in data from a URL
hallmarks_url <- "https://data.broadinstitute.org/gsea-msigdb/msigdb/release/7.1/h.all.v7.1.symbols.gmt"

# QuSAGE is another pathway analysis method, the qusage package has a function
# for reading GMT files and turning them into a list
hallmarks_list <- qusage::read.gmt(hallmarks_url)

What does this list look like?

head(hallmarks_list)

RNA-seq data

We have VST transformed RNA-seq data, which is on a log2-like scale. These data are from the Open Pediatric Brain Tumor Atlas (OpenPBTA) OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children’s Hospital of Philadelphia conducted openly on GitHub.

You can read more about the project here.

We’re only working with the medulloblastoma samples in this example.

rnaseq_df <- readr::read_tsv(rnaseq_file)

── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────
cols(
  .default = col_double(),
  gene_symbol = col_character()
)
ℹ Use `spec()` for the full column specifications.
# What does the RNA-seq data frame look like?
rnaseq_df[1:5, 1:5]

For GSVA, we need a matrix.

rnaseq_mat <- rnaseq_df %>%
  tibble::column_to_rownames("gene_symbol") %>%
  as.matrix()

Note: If we had duplicate gene symbols here, we couldn’t set them as rownames.

GSVA

Figure 1 from Hänzelmann et al. (2013).

You may notice that GSVA has some commonalities with GSEA. Rather than ranking genes based on some statistic we selected ahead of time, GSVA fits a model and ranks genes based on their expression level relative to the sample distribution. This is a way of asking if a gene i is highly or lowly expressed in a sample j in the context of this experiment and ranking accordingly (Hänzelmann et al. 2013). The pathway-level score calculated is a way of asking how genes within a gene set vary as compared to genes that are outside of that gene set (Malhotra. 2018). (This is sometimes called a competitive test in gene set enrichment literature.) The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) (Hänzelmann et al. 2013).

The output is a gene set by sample matrix of GSVA scores.

Perform GSVA

gsva_results <- gsva(rnaseq_mat, 
                     hallmarks_list,
                     method = "gsva",
                     # Appropriate for our transformed data on log2-like scale
                     kcdf = "Gaussian",  
                     # Minimum gene set size
                     min.sz = 15, 
                     # Maximum gene set size
                     max.sz = 500,
                     # Compute Gaussian-distributed scores
                     mx.diff = TRUE)
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 50 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                                          
  |                                                                                                                    |   0%
  |                                                                                                                          
  |==                                                                                                                  |   2%
  |                                                                                                                          
  |=====                                                                                                               |   4%
  |                                                                                                                          
  |=======                                                                                                             |   6%
  |                                                                                                                          
  |=========                                                                                                           |   8%
  |                                                                                                                          
  |============                                                                                                        |  10%
  |                                                                                                                          
  |==============                                                                                                      |  12%
  |                                                                                                                          
  |================                                                                                                    |  14%
  |                                                                                                                          
  |===================                                                                                                 |  16%
  |                                                                                                                          
  |=====================                                                                                               |  18%
  |                                                                                                                          
  |=======================                                                                                             |  20%
  |                                                                                                                          
  |==========================                                                                                          |  22%
  |                                                                                                                          
  |============================                                                                                        |  24%
  |                                                                                                                          
  |==============================                                                                                      |  26%
  |                                                                                                                          
  |================================                                                                                    |  28%
  |                                                                                                                          
  |===================================                                                                                 |  30%
  |                                                                                                                          
  |=====================================                                                                               |  32%
  |                                                                                                                          
  |=======================================                                                                             |  34%
  |                                                                                                                          
  |==========================================                                                                          |  36%
  |                                                                                                                          
  |============================================                                                                        |  38%
  |                                                                                                                          
  |==============================================                                                                      |  40%
  |                                                                                                                          
  |=================================================                                                                   |  42%
  |                                                                                                                          
  |===================================================                                                                 |  44%
  |                                                                                                                          
  |=====================================================                                                               |  46%
  |                                                                                                                          
  |========================================================                                                            |  48%
  |                                                                                                                          
  |==========================================================                                                          |  50%
  |                                                                                                                          
  |============================================================                                                        |  52%
  |                                                                                                                          
  |===============================================================                                                     |  54%
  |                                                                                                                          
  |=================================================================                                                   |  56%
  |                                                                                                                          
  |===================================================================                                                 |  58%
  |                                                                                                                          
  |======================================================================                                              |  60%
  |                                                                                                                          
  |========================================================================                                            |  62%
  |                                                                                                                          
  |==========================================================================                                          |  64%
  |                                                                                                                          
  |=============================================================================                                       |  66%
  |                                                                                                                          
  |===============================================================================                                     |  68%
  |                                                                                                                          
  |=================================================================================                                   |  70%
  |                                                                                                                          
  |====================================================================================                                |  72%
  |                                                                                                                          
  |======================================================================================                              |  74%
  |                                                                                                                          
  |========================================================================================                            |  76%
  |                                                                                                                          
  |==========================================================================================                          |  78%
  |                                                                                                                          
  |=============================================================================================                       |  80%
  |                                                                                                                          
  |===============================================================================================                     |  82%
  |                                                                                                                          
  |=================================================================================================                   |  84%
  |                                                                                                                          
  |====================================================================================================                |  86%
  |                                                                                                                          
  |======================================================================================================              |  88%
  |                                                                                                                          
  |========================================================================================================            |  90%
  |                                                                                                                          
  |===========================================================================================================         |  92%
  |                                                                                                                          
  |=============================================================================================================       |  94%
  |                                                                                                                          
  |===============================================================================================================     |  96%
  |                                                                                                                          
  |==================================================================================================================  |  98%
  |                                                                                                                          
  |====================================================================================================================| 100%

Note: the gsva() documentation says we can use kcdf = "Gaussian" if we had RNA-seq log-CPMs, log-RPKMs or log-TPMs, but we would use kcdf = "Poisson" on integer counts.

# Let's explore what the output of gsva() looks like
gsva_results[1:5, 1:5]
                                    BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT BS_1QXEC43H BS_1TWCV047
HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193  0.32300630  -0.1468081
HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759  0.36247083  -0.2971559
HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933  0.32418657  -0.4561386
HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076 -0.07023068  -0.1861483
HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594  0.41372523  -0.1152437

A note on gene set size

Often the scores of gene set enrichment methods are not comparable between gene sets of different sizes. Let’s do an experiment using randomly generated gene sets to explore this idea a bit more.

We need to get a collection of all possible genes we will sample from to create random gene sets. Because we’re doing some random sampling, we need to set a seed for this to be reproducible.

# Use all the gene symbols in the dataset as the pool of possible genes
all_genes <- rownames(rnaseq_mat)

# Set a seed for reproducibility
set.seed(2020)

Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We’ll use the same minimum and maximum values for our random gene sets and some values in between.

# Make a list of integers that indicate the random gene set sizes
gene_set_size <- list(15, 25, 50, 100, 250, 500)

For each gene set size, we will generate 100 random gene sets.

# Set number of replicates
nreps <- 100
# Generate 100 random gene sets of each size
random_gene_sets <- rep(gene_set_size, nreps) %>%  # Repeat gene sizes so we run `nreps` times 
  purrr::map(
    # Sample the vector of all genes, choosing the number of items specified
    # in the element of gene set size
    ~ base::sample(x = all_genes,
                   size = .x)
  )

The Hallmarks list we used earlier stored the gene set names as the name of the list, so let’s add names to our random gene sets that indicate what size they are and so gsva() doesn’t get upset.

# We will include the size of the gene set in the gene set name 
# Start by taking the length of each pathway and appending "pathway_" to that
# number
lengths_vector <- random_gene_sets %>%
  # Get the length of each gene set (number of genes)
  purrr::map(~ length(.x)) %>%
  # Make it "pathway_<gene set size>"
  purrr::map(~ paste0("pathway_", .x)) %>%
  # Return a vector
  purrr::flatten_chr()

# Add the names in lengths_vector to the list - "pathway_<gene set size>"
random_gene_sets <- random_gene_sets %>% 
  # make.names() appends a "version" if something is not unique
  purrr::set_names(nm = make.names(lengths_vector, unique = TRUE))

Run GSVA on our dataset with the same parameters as before, but now with random gene sets.

random_gsva_results <- gsva(rnaseq_mat, 
                            random_gene_sets,
                            method = "gsva",
                            # Appropriate for our transformed data on 
                            # log2-like scale
                            kcdf = "Gaussian",  
                            # Minimum gene set size
                            min.sz = 15, 
                            # Maximum gene set size
                            max.sz = 500,
                            # Compute Gaussian-distributed scores
                            mx.diff = TRUE)                            
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 579 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                                          
  |                                                                                                                    |   0%
  |                                                                                                                          
  |=                                                                                                                   |   1%
  |                                                                                                                          
  |==                                                                                                                  |   1%
  |                                                                                                                          
  |==                                                                                                                  |   2%
  |                                                                                                                          
  |===                                                                                                                 |   2%
  |                                                                                                                          
  |===                                                                                                                 |   3%
  |                                                                                                                          
  |====                                                                                                                |   3%
  |                                                                                                                          
  |====                                                                                                                |   4%
  |                                                                                                                          
  |=====                                                                                                               |   4%
  |                                                                                                                          
  |=====                                                                                                               |   5%
  |                                                                                                                          
  |======                                                                                                              |   5%
  |                                                                                                                          
  |======                                                                                                              |   6%
  |                                                                                                                          
  |=======                                                                                                             |   6%
  |                                                                                                                          
  |========                                                                                                            |   7%
  |                                                                                                                          
  |=========                                                                                                           |   7%
  |                                                                                                                          
  |=========                                                                                                           |   8%
  |                                                                                                                          
  |==========                                                                                                          |   8%
  |                                                                                                                          
  |==========                                                                                                          |   9%
  |                                                                                                                          
  |===========                                                                                                         |   9%
  |                                                                                                                          
  |===========                                                                                                         |  10%
  |                                                                                                                          
  |============                                                                                                        |  10%
  |                                                                                                                          
  |============                                                                                                        |  11%
  |                                                                                                                          
  |=============                                                                                                       |  11%
  |                                                                                                                          
  |=============                                                                                                       |  12%
  |                                                                                                                          
  |==============                                                                                                      |  12%
  |                                                                                                                          
  |===============                                                                                                     |  13%
  |                                                                                                                          
  |================                                                                                                    |  13%
  |                                                                                                                          
  |================                                                                                                    |  14%
  |                                                                                                                          
  |=================                                                                                                   |  14%
  |                                                                                                                          
  |=================                                                                                                   |  15%
  |                                                                                                                          
  |==================                                                                                                  |  15%
  |                                                                                                                          
  |==================                                                                                                  |  16%
  |                                                                                                                          
  |===================                                                                                                 |  16%
  |                                                                                                                          
  |===================                                                                                                 |  17%
  |                                                                                                                          
  |====================                                                                                                |  17%
  |                                                                                                                          
  |====================                                                                                                |  18%
  |                                                                                                                          
  |=====================                                                                                               |  18%
  |                                                                                                                          
  |======================                                                                                              |  19%
  |                                                                                                                          
  |=======================                                                                                             |  20%
  |                                                                                                                          
  |========================                                                                                            |  20%
  |                                                                                                                          
  |========================                                                                                            |  21%
  |                                                                                                                          
  |=========================                                                                                           |  21%
  |                                                                                                                          
  |=========================                                                                                           |  22%
  |                                                                                                                          
  |==========================                                                                                          |  22%
  |                                                                                                                          
  |==========================                                                                                          |  23%
  |                                                                                                                          
  |===========================                                                                                         |  23%
  |                                                                                                                          
  |===========================                                                                                         |  24%
  |                                                                                                                          
  |============================                                                                                        |  24%
  |                                                                                                                          
  |============================                                                                                        |  25%
  |                                                                                                                          
  |=============================                                                                                       |  25%
  |                                                                                                                          
  |==============================                                                                                      |  26%
  |                                                                                                                          
  |===============================                                                                                     |  26%
  |                                                                                                                          
  |===============================                                                                                     |  27%
  |                                                                                                                          
  |================================                                                                                    |  27%
  |                                                                                                                          
  |================================                                                                                    |  28%
  |                                                                                                                          
  |=================================                                                                                   |  28%
  |                                                                                                                          
  |=================================                                                                                   |  29%
  |                                                                                                                          
  |==================================                                                                                  |  29%
  |                                                                                                                          
  |==================================                                                                                  |  30%
  |                                                                                                                          
  |===================================                                                                                 |  30%
  |                                                                                                                          
  |===================================                                                                                 |  31%
  |                                                                                                                          
  |====================================                                                                                |  31%
  |                                                                                                                          
  |=====================================                                                                               |  32%
  |                                                                                                                          
  |======================================                                                                              |  32%
  |                                                                                                                          
  |======================================                                                                              |  33%
  |                                                                                                                          
  |=======================================                                                                             |  33%
  |                                                                                                                          
  |=======================================                                                                             |  34%
  |                                                                                                                          
  |========================================                                                                            |  34%
  |                                                                                                                          
  |========================================                                                                            |  35%
  |                                                                                                                          
  |=========================================                                                                           |  35%
  |                                                                                                                          
  |=========================================                                                                           |  36%
  |                                                                                                                          
  |==========================================                                                                          |  36%
  |                                                                                                                          
  |==========================================                                                                          |  37%
  |                                                                                                                          
  |===========================================                                                                         |  37%
  |                                                                                                                          
  |============================================                                                                        |  38%
  |                                                                                                                          
  |=============================================                                                                       |  39%
  |                                                                                                                          
  |==============================================                                                                      |  39%
  |                                                                                                                          
  |==============================================                                                                      |  40%
  |                                                                                                                          
  |===============================================                                                                     |  40%
  |                                                                                                                          
  |===============================================                                                                     |  41%
  |                                                                                                                          
  |================================================                                                                    |  41%
  |                                                                                                                          
  |================================================                                                                    |  42%
  |                                                                                                                          
  |=================================================                                                                   |  42%
  |                                                                                                                          
  |=================================================                                                                   |  43%
  |                                                                                                                          
  |==================================================                                                                  |  43%
  |                                                                                                                          
  |==================================================                                                                  |  44%
  |                                                                                                                          
  |===================================================                                                                 |  44%
  |                                                                                                                          
  |====================================================                                                                |  45%
  |                                                                                                                          
  |=====================================================                                                               |  45%
  |                                                                                                                          
  |=====================================================                                                               |  46%
  |                                                                                                                          
  |======================================================                                                              |  46%
  |                                                                                                                          
  |======================================================                                                              |  47%
  |                                                                                                                          
  |=======================================================                                                             |  47%
  |                                                                                                                          
  |=======================================================                                                             |  48%
  |                                                                                                                          
  |========================================================                                                            |  48%
  |                                                                                                                          
  |========================================================                                                            |  49%
  |                                                                                                                          
  |=========================================================                                                           |  49%
  |                                                                                                                          
  |=========================================================                                                           |  50%
  |                                                                                                                          
  |==========================================================                                                          |  50%
  |                                                                                                                          
  |===========================================================                                                         |  50%
  |                                                                                                                          
  |===========================================================                                                         |  51%
  |                                                                                                                          
  |============================================================                                                        |  51%
  |                                                                                                                          
  |============================================================                                                        |  52%
  |                                                                                                                          
  |=============================================================                                                       |  52%
  |                                                                                                                          
  |=============================================================                                                       |  53%
  |                                                                                                                          
  |==============================================================                                                      |  53%
  |                                                                                                                          
  |==============================================================                                                      |  54%
  |                                                                                                                          
  |===============================================================                                                     |  54%
  |                                                                                                                          
  |===============================================================                                                     |  55%
  |                                                                                                                          
  |================================================================                                                    |  55%
  |                                                                                                                          
  |=================================================================                                                   |  56%
  |                                                                                                                          
  |==================================================================                                                  |  56%
  |                                                                                                                          
  |==================================================================                                                  |  57%
  |                                                                                                                          
  |===================================================================                                                 |  57%
  |                                                                                                                          
  |===================================================================                                                 |  58%
  |                                                                                                                          
  |====================================================================                                                |  58%
  |                                                                                                                          
  |====================================================================                                                |  59%
  |                                                                                                                          
  |=====================================================================                                               |  59%
  |                                                                                                                          
  |=====================================================================                                               |  60%
  |                                                                                                                          
  |======================================================================                                              |  60%
  |                                                                                                                          
  |======================================================================                                              |  61%
  |                                                                                                                          
  |=======================================================================                                             |  61%
  |                                                                                                                          
  |========================================================================                                            |  62%
  |                                                                                                                          
  |=========================================================================                                           |  63%
  |                                                                                                                          
  |==========================================================================                                          |  63%
  |                                                                                                                          
  |==========================================================================                                          |  64%
  |                                                                                                                          
  |===========================================================================                                         |  64%
  |                                                                                                                          
  |===========================================================================                                         |  65%
  |                                                                                                                          
  |============================================================================                                        |  65%
  |                                                                                                                          
  |============================================================================                                        |  66%
  |                                                                                                                          
  |=============================================================================                                       |  66%
  |                                                                                                                          
  |=============================================================================                                       |  67%
  |                                                                                                                          
  |==============================================================================                                      |  67%
  |                                                                                                                          
  |==============================================================================                                      |  68%
  |                                                                                                                          
  |===============================================================================                                     |  68%
  |                                                                                                                          
  |================================================================================                                    |  69%
  |                                                                                                                          
  |=================================================================================                                   |  69%
  |                                                                                                                          
  |=================================================================================                                   |  70%
  |                                                                                                                          
  |==================================================================================                                  |  70%
  |                                                                                                                          
  |==================================================================================                                  |  71%
  |                                                                                                                          
  |===================================================================================                                 |  71%
  |                                                                                                                          
  |===================================================================================                                 |  72%
  |                                                                                                                          
  |====================================================================================                                |  72%
  |                                                                                                                          
  |====================================================================================                                |  73%
  |                                                                                                                          
  |=====================================================================================                               |  73%
  |                                                                                                                          
  |=====================================================================================                               |  74%
  |                                                                                                                          
  |======================================================================================                              |  74%
  |                                                                                                                          
  |=======================================================================================                             |  75%
  |                                                                                                                          
  |========================================================================================                            |  75%
  |                                                                                                                          
  |========================================================================================                            |  76%
  |                                                                                                                          
  |=========================================================================================                           |  76%
  |                                                                                                                          
  |=========================================================================================                           |  77%
  |                                                                                                                          
  |==========================================================================================                          |  77%
  |                                                                                                                          
  |==========================================================================================                          |  78%
  |                                                                                                                          
  |===========================================================================================                         |  78%
  |                                                                                                                          
  |===========================================================================================                         |  79%
  |                                                                                                                          
  |============================================================================================                        |  79%
  |                                                                                                                          
  |============================================================================================                        |  80%
  |                                                                                                                          
  |=============================================================================================                       |  80%
  |                                                                                                                          
  |==============================================================================================                      |  81%
  |                                                                                                                          
  |===============================================================================================                     |  82%
  |                                                                                                                          
  |================================================================================================                    |  82%
  |                                                                                                                          
  |================================================================================================                    |  83%
  |                                                                                                                          
  |=================================================================================================                   |  83%
  |                                                                                                                          
  |=================================================================================================                   |  84%
  |                                                                                                                          
  |==================================================================================================                  |  84%
  |                                                                                                                          
  |==================================================================================================                  |  85%
  |                                                                                                                          
  |===================================================================================================                 |  85%
  |                                                                                                                          
  |===================================================================================================                 |  86%
  |                                                                                                                          
  |====================================================================================================                |  86%
  |                                                                                                                          
  |====================================================================================================                |  87%
  |                                                                                                                          
  |=====================================================================================================               |  87%
  |                                                                                                                          
  |======================================================================================================              |  88%
  |                                                                                                                          
  |=======================================================================================================             |  88%
  |                                                                                                                          
  |=======================================================================================================             |  89%
  |                                                                                                                          
  |========================================================================================================            |  89%
  |                                                                                                                          
  |========================================================================================================            |  90%
  |                                                                                                                          
  |=========================================================================================================           |  90%
  |                                                                                                                          
  |=========================================================================================================           |  91%
  |                                                                                                                          
  |==========================================================================================================          |  91%
  |                                                                                                                          
  |==========================================================================================================          |  92%
  |                                                                                                                          
  |===========================================================================================================         |  92%
  |                                                                                                                          
  |===========================================================================================================         |  93%
  |                                                                                                                          
  |============================================================================================================        |  93%
  |                                                                                                                          
  |=============================================================================================================       |  94%
  |                                                                                                                          
  |==============================================================================================================      |  94%
  |                                                                                                                          
  |==============================================================================================================      |  95%
  |                                                                                                                          
  |===============================================================================================================     |  95%
  |                                                                                                                          
  |===============================================================================================================     |  96%
  |                                                                                                                          
  |================================================================================================================    |  96%
  |                                                                                                                          
  |================================================================================================================    |  97%
  |                                                                                                                          
  |=================================================================================================================   |  97%
  |                                                                                                                          
  |=================================================================================================================   |  98%
  |                                                                                                                          
  |==================================================================================================================  |  98%
  |                                                                                                                          
  |==================================================================================================================  |  99%
  |                                                                                                                          
  |=================================================================================================================== |  99%
  |                                                                                                                          
  |====================================================================================================================| 100%

Now let’s make a plot to look at the distribution of scores from random gene sets. First we need to get this data in an appropriate format for ggplot2.

# The random results are a matrix
random_long_df <- random_gsva_results %>%
  data.frame() %>%
  # Gene set names are rownames
  tibble::rownames_to_column("gene_set") %>%
  # Get into long format
  tidyr::pivot_longer(cols = -gene_set, 
                      names_to = "Kids_First_Biospecimen_ID",
                      values_to = "gsva_score") %>%
  # Remove the .version added by make.names()
  dplyr::mutate(gene_set = stringr::str_remove(gene_set, "\\..*")) %>%
  # Add a column that keeps track of the gene set size
  dplyr::mutate(gene_set_size = stringr::word(gene_set, 2, sep = "_")) %>%
  # We want to plot smallest no. genes -> largest no. genes
  dplyr::mutate(gene_set_size = factor(gene_set_size, 
                                       levels = c(15, 25, 50, 100, 250, 500)))

Let’s make a violin plot so we can look at the distribution of scores by gene set size.

# Violin plot comparing GSVA scores of different random gene set sizes
random_long_df %>%
  ggplot2::ggplot(ggplot2::aes(x = gene_set_size, 
                               y = gsva_score)) +
  # Make a violin plot that's a pretty blue!
  ggplot2::geom_violin(fill = "#99CCFF", alpha = 0.5) +
  # Add a point with the mean value
  ggplot2::stat_summary(
    geom = "point",
    fun = "mean",
    # Change the aesthetics of the points
    size = 3,
    color = "#0066CC",
    shape = 18
  ) +
  # Flip the axes
  ggplot2::coord_flip() +
  ggplot2::labs(title = "Random Gene Set GSVA Scores",
                x = "gene set size",
                y = "GSVA score") +
  ggplot2::theme_bw()

What do you notice about these distributions? How might you use this information to inform your interpretation of GSVA scores?

How can you use these scores?

If you did have groups information for your samples, you could test for pathway scores differences between groups. Here’s an example of that from the OpenPBTA project itself!

You can also visualize this matrix in a heatmap. Here’s a figure from the OpenPBTA project, where the middle panel is a heatmap of GSVA scores that were significantly different between histologies.

Code here.

Write results to file

gsva_results %>%
  as.data.frame() %>%
  tibble::rownames_to_column("pathway") %>%
  readr::write_tsv(file = gsva_results_file)

Session Info

sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
 [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] GSVA_1.38.2    magrittr_2.0.1

loaded via a namespace (and not attached):
 [1] MatrixGenerics_1.2.0        Biobase_2.50.0              httr_1.4.2                  tidyr_1.1.2                
 [5] bit64_4.0.5                 jsonlite_1.7.2              assertthat_0.2.1            BiocManager_1.30.10        
 [9] stats4_4.0.3                blob_1.2.1                  renv_0.12.5-2               GenomeInfoDbData_1.2.4     
[13] pillar_1.4.7                RSQLite_2.2.3               lattice_0.20-41             glue_1.4.2                 
[17] limma_3.46.0                digest_0.6.27               GenomicRanges_1.42.0        XVector_0.30.0             
[21] colorspace_2.0-0            Matrix_1.3-2                GSEABase_1.52.1             XML_3.99-0.5               
[25] pkgconfig_2.0.3             zlibbioc_1.36.0             purrr_0.3.4                 xtable_1.8-4               
[29] mvtnorm_1.1-1               scales_1.1.1                BiocParallel_1.24.1         emmeans_1.5.3              
[33] tibble_3.0.5                annotate_1.68.0             farver_2.0.3                generics_0.1.0             
[37] IRanges_2.24.1              ggplot2_3.3.3               ellipsis_0.3.1              SummarizedExperiment_1.20.0
[41] BiocGenerics_0.36.0         cli_2.2.0                   crayon_1.3.4                memoise_1.1.0              
[45] estimability_1.3            fansi_0.4.2                 nlme_3.1-151                graph_1.68.0               
[49] tools_4.0.3                 hms_1.0.0                   lifecycle_0.2.0             matrixStats_0.57.0         
[53] stringr_1.4.0               S4Vectors_0.28.1            fftw_1.0-6                  munsell_0.5.0              
[57] DelayedArray_0.16.2         AnnotationDbi_1.52.0        compiler_4.0.3              GenomeInfoDb_1.26.2        
[61] rlang_0.4.10                grid_4.0.3                  RCurl_1.98-1.2              rstudioapi_0.13            
[65] labeling_0.4.2              bitops_1.0-6                qusage_2.24.0               gtable_0.3.0               
[69] DBI_1.1.1                   R6_2.5.0                    knitr_1.30                  dplyr_1.0.3                
[73] bit_4.0.4                   readr_1.4.0                 stringi_1.5.3               parallel_4.0.3             
[77] Rcpp_1.0.6                  vctrs_0.3.6                 tidyselect_1.1.0            xfun_0.20                  
[81] coda_0.19-4                
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSIKb3V0cHV0OiAKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjAKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBJZGVudGlmeSB3aGVuIEdlbmUgU2V0IFZhcmlhdGlvbiBBbmFseXNpcyAoR1NWQSkgaXMgd2VsbC1zdWl0ZWQgZm9yIGFuIGFuYWx5c2lzCi0gUGVyZm9ybSBHU1ZBIG9uIHRyYW5zZm9ybWVkIFJOQS1zZXEgZGF0YSB3aXRoIHRoZSBgR1NWQWAgcGFja2FnZQotIEV4cGxvcmUgdGhlIGRlcGVuZGVuY2Ugb2YgR1NWQSBzY29yZXMgb24gZ2VuZSBzZXQgc2l6ZSB3aXRoIHJhbmRvbSBnZW5lIHNldHMKCi0tLQoKU28gZmFyIGV2ZXJ5IHBhdGh3YXkgYW5hbHlzaXMgbWV0aG9kIHdlJ3ZlIGNvdmVyZWQgcmVsaWVzIG9uIHNvbWUgaW5mb3JtYXRpb24gYWJvdXQgZ3JvdXBzIG9mIHNhbXBsZXMgaW4gb3VyIGRhdGEuCkZvciBvdmVyLXJlcHJlc2VudGF0aW9uIGFuYWx5c2lzIChPUkEpLCB3ZSBjcmVhdGVkIGdlbmUgc2V0cyBmcm9tIHR3byBkaWZmZXJlbnQgdHdvIGdyb3VwIGNvbXBhcmlzb25zLgpJbiB0aGUgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkgZXhhbXBsZSwgd2UgdXNlZCBzdGF0aXN0aWNzIGZyb20gYSBkaWZmZXJlbnRpYWwgZ2VuZSBleHByZXNzaW9uIChER0UpIGFuYWx5c2lzIHdoZXJlIHdlIGNvbXBhcmVkIF9NWUNOXyBhbXBsaWZpZWQgY2VsbCBsaW5lcyB0byBub24tYW1wbGlmaWVkIGNlbGwgbGluZXM7IHdlIG5lZWRlZCB0aGF0IGFtcGxpZmljYXRpb24gc3RhdHVzIGluZm9ybWF0aW9uLgoKV2hhdCBpZiB3ZSdyZSBsZXNzIHN1cmUgYWJvdXQgZ3JvdXBzIGluIG91ciBkYXRhIG9yIHdlIHdhbnQgdG8gYW5hbHl6ZSBvdXIgZGF0YSBpbiBhIG1vcmUgdW5zdXBlcnZpc2VkIG1hbm5lcj8KCkluIHRoaXMgbm90ZWJvb2sgd2Ugd2lsbCBjb3ZlciBhIG1ldGhvZCBjYWxsZWQgR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSAoW0jDpG56ZWxtYW5uIF9ldCBhbC5fIDIwMTNdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2LzE0NzEtMjEwNS0xNC03KSkgdGhhdCBhbGxvd3MgdXMgdG8gY2FsY3VsYXRlIGdlbmUgc2V0IG9yIHBhdGh3YXkgc2NvcmVzIG9uIGEgcGVyLXNhbXBsZSBiYXNpcy4KCldlIGxpa2UgdGhpcyBxdW90ZSBmcm9tIHRoZSBHU1ZBIHBhcGVyIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKSB0byBzZXQgdGhlIHN0YWdlOgoKPiBXaGlsZSBbZ2VuZSBzZXQgZW5yaWNobWVudF0gbWV0aG9kcyBhcmUgZ2VuZXJhbGx5IHJlZ2FyZGVkIGFzIGVuZCBwb2ludHMgb2YgYSBiaW9pbmZvcm1hdGljIGFuYWx5c2lzLCBHU1ZBIGNvbnN0aXR1dGVzIGEgc3RhcnRpbmcgcG9pbnQgdG8gYnVpbGQgcGF0aHdheS1jZW50cmljIG1vZGVscyBvZiBiaW9sb2d5LgoKUmF0aGVyIHRoYW4gY29udGV4dHVhbGl6aW5nIHNvbWUgcmVzdWx0cyB5b3UgX2FscmVhZHkgaGF2ZV8gZnJvbSBhbm90aGVyIGFuYWx5c2lzIGxpa2UgREdFLCBHU1ZBIGlzIGRlc2lnbmVkIHRvIHByb3ZpZGUgYW4gZXN0aW1hdGUgb2YgcGF0aHdheSB2YXJpYXRpb24gZm9yIGVhY2ggb2YgdGhlIHNhbXBsZXMgaW4gYW4gZXhwZXJpbWVudC4KTm90ZSB0aGF0IHRoZXNlIHNjb3JlcyB3aWxsIGRlcGVuZCBvbiB0aGUgc2FtcGxlcyBpbmNsdWRlZCBpbiB0aGUgZGF0YXNldCB3aGVuIHlvdSBydW4gR1NWQTsgaWYgeW91IGFkZGVkIG1vcmUgc2FtcGxlcyBhbmQgcmVyYW4gR1NWQSwgeW91IHdvdWxkIGV4cGVjdCB0aGUgc2NvcmVzIHRvIGNoYW5nZS4KCiMjIyMgT3RoZXIgcmVzb3VyY2VzCgoqIFtNYWxob3RyYS4gX0RlY29kaW5nIEdlbmUgU2V0IFZhcmlhdGlvbiBBbmFseXNpc18uICgyMDE4KV0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL2RlY29kaW5nLWdlbmUtc2V0LXZhcmlhdGlvbi1hbmFseXNpcy04MTkzYTBjZmRhMykKCiMjIFNldCB1cAoKIyMjIExpYnJhcmllcwoKYGBge3IgbGlicmFyaWVzfQojIFBpcGVzCmxpYnJhcnkobWFncml0dHIpCiMgR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzCmxpYnJhcnkoR1NWQSkKYGBgCgojIyMgRGlyZWN0b3JpZXMgYW5kIGZpbGVzCgojIyMjIERpcmVjdG9yaWVzCgpgYGB7ciBkaXJlY3Rvcmllc30KIyBXZSBoYXZlIHNvbWUgbWVkdWxsb2JsYXN0b21hIGRhdGEgZnJvbSB0aGUgT3BlblBCVEEgcHJvamVjdCB0aGF0IHdlJ3ZlIAojIHByZXBhcmVkIGFoZWFkIG9mIHRpbWUKaW5wdXRfZGlyIDwtIGZpbGUucGF0aCgiZGF0YSIsICJvcGVuLXBidGEiKQoKIyBDcmVhdGUgYSBkaXJlY3Rvcnkgc3BlY2lmaWNhbGx5IGZvciB0aGUgcmVzdWx0cyB1c2luZyB0aGlzIGRhdGFzZXQKb3V0cHV0X2RpciA8LSBmaWxlLnBhdGgoInJlc3VsdHMiLCAib3Blbi1wYnRhIikgCmlmICghZGlyLmV4aXN0cyhvdXRwdXRfZGlyKSkgewogIGRpci5jcmVhdGUob3V0cHV0X2RpciwgcmVjdXJzaXZlID0gVFJVRSkKfQpgYGAKCiMjIyMgSW5wdXQKCldlIGhhdmUgVlNUIHRyYW5zZm9ybWVkIFJOQS1zZXEgZGF0YSwgYW5ub3RhdGVkIHdpdGggZ2VuZSBzeW1ib2xzLCB0aGF0IGhhcyBiZWVuIGNvbGxhcHNlZCBzdWNoIHRoYXQgdGhlcmUgYXJlIG5vIGR1cGxpY2F0ZWQgZ2VuZSBpZGVudGlmaWVycyAoc2VlIGBzZXR1cGApLgoKYGBge3IgaW5wdXRfZmlsZX0Kcm5hc2VxX2ZpbGUgPC0gZmlsZS5wYXRoKGlucHV0X2RpciwgIm1lZHVsbG9ibGFzdG9tYV92c3RfY29sbGFwc2VkLnRzdiIpCmBgYAoKIyMjIyBPdXRwdXQKCmBgYHtyIG91dHB1dF9maWxlLCBsaXZlID0gVFJVRX0KZ3N2YV9yZXN1bHRzX2ZpbGUgPC0gZmlsZS5wYXRoKG91dHB1dF9kaXIsICJtZWR1bGxvYmxhc3RvbWFfZ3N2YV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMgR2VuZSBzZXRzCgpUaGUgZnVuY3Rpb24gdGhhdCB3ZSB3aWxsIHVzZSB0byBydW4gR1NWQSB3YW50cyB0aGUgZ2VuZSBzZXRzIHRvIGJlIGluIGEgbGlzdCwgcmF0aGVyIHRoYW4gYSB0aWR5IGRhdGEgZnJhbWUgdGhhdCB3ZSB1c2VkIHdpdGggYGNsdXN0ZXJQcm9maWxlcmAgKGFsdGhvdWdoIGl0IGRvZXMgYWNjZXB0IG90aGVyIGZvcm1hdHMpLgoKV2UncmUgZ29pbmcgdG8gdGFrZSB0aGlzIG9wcG9ydHVuaXR5IHRvIGludHJvZHVjZSBhIGRpZmZlcmVudCBmb3JtYXQgdGhhdCBnZW5lIHNldHMgYXJlIG9mdGVuIGRpc3RyaWJ1dGVkIGluIGNhbGxlZCBbR01UIChHZW5lIE1hdHJpeCBUcmFuc3Bvc2VkKV0oaHR0cHM6Ly9zb2Z0d2FyZS5icm9hZGluc3RpdHV0ZS5vcmcvY2FuY2VyL3NvZnR3YXJlL2dzZWEvd2lraS9pbmRleC5waHAvRGF0YV9mb3JtYXRzI0dNVDpfR2VuZV9NYXRyaXhfVHJhbnNwb3NlZF9maWxlX2Zvcm1hdF8uMjguMkEuZ210LjI5KS4KCldlJ3JlIGdvaW5nIHRvIHJlYWQgaW4gdGhlIEhhbGxtYXJrIGNvbGxlY3Rpb24gZmlsZSBkaXJlY3RseSBmcm9tIFtNU2lnREJdKGh0dHBzOi8vd3d3LmdzZWEtbXNpZ2RiLm9yZy9nc2VhL21zaWdkYi9pbmRleC5qc3ApLCByYXRoZXIgdGhhbiB1c2luZyBgbXNpZ2RicmAgbGlrZSB3ZSBkaWQgaW4gZWFybGllciBub3RlYm9va3MuCgpUaGUgUk5BLXNlcSBkYXRhIHVzZXMgZ2VuZSBzeW1ib2xzLCBzbyB3ZSBuZWVkIGdlbmUgc2V0cyB0aGF0IHVzZSBnZW5lIHN5bWJvbHMsIHRvby4KCmBgYHtyIGdtdH0KIyBSIGNhbiBvZnRlbiByZWFkIGluIGRhdGEgZnJvbSBhIFVSTApoYWxsbWFya3NfdXJsIDwtICJodHRwczovL2RhdGEuYnJvYWRpbnN0aXR1dGUub3JnL2dzZWEtbXNpZ2RiL21zaWdkYi9yZWxlYXNlLzcuMS9oLmFsbC52Ny4xLnN5bWJvbHMuZ210IgoKIyBRdVNBR0UgaXMgYW5vdGhlciBwYXRod2F5IGFuYWx5c2lzIG1ldGhvZCwgdGhlIHF1c2FnZSBwYWNrYWdlIGhhcyBhIGZ1bmN0aW9uCiMgZm9yIHJlYWRpbmcgR01UIGZpbGVzIGFuZCB0dXJuaW5nIHRoZW0gaW50byBhIGxpc3QKaGFsbG1hcmtzX2xpc3QgPC0gcXVzYWdlOjpyZWFkLmdtdChoYWxsbWFya3NfdXJsKQpgYGAKCldoYXQgZG9lcyB0aGlzIGxpc3QgbG9vayBsaWtlPwoKYGBge3IgaGVhZF9oYWxsbWFyaywgZXZhbCA9IEZBTFNFfQpoZWFkKGhhbGxtYXJrc19saXN0KQpgYGAKCiMjIFJOQS1zZXEgZGF0YQoKV2UgaGF2ZSBWU1QgdHJhbnNmb3JtZWQgUk5BLXNlcSBkYXRhLCB3aGljaCBpcyBvbiBhIGxvZzItbGlrZSBzY2FsZS4KVGhlc2UgZGF0YSBhcmUgZnJvbSB0aGUgT3BlbiBQZWRpYXRyaWMgQnJhaW4gVHVtb3IgQXRsYXMgKE9wZW5QQlRBKQpPcGVuUEJUQSBpcyBhIGNvbGxhYm9yYXRpdmUgcHJvamVjdCBvcmdhbml6ZWQgYnkgdGhlIENDREwgYW5kIHRoZSBDZW50ZXIgZm9yIERhdGEtRHJpdmVuIERpc2NvdmVyeSBpbiBCaW9tZWRpY2luZSAoRDNiKSBhdCB0aGUgQ2hpbGRyZW4ncyBIb3NwaXRhbCBvZiBQaGlsYWRlbHBoaWEgY29uZHVjdGVkIG9wZW5seSBvbiBHaXRIdWIuCgpZb3UgY2FuIHJlYWQgbW9yZSBhYm91dCB0aGUgcHJvamVjdCBbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL2FsZXhzbGVtb25hZGUvb3BlbnBidGEtYW5hbHlzaXMvI29wZW5wYnRhLWFuYWx5c2lzKS4KCldlJ3JlIG9ubHkgd29ya2luZyB3aXRoIHRoZSBtZWR1bGxvYmxhc3RvbWEgc2FtcGxlcyBpbiB0aGlzIGV4YW1wbGUuCgpgYGB7ciByZWFkX2luX3JuYXNlcX0Kcm5hc2VxX2RmIDwtIHJlYWRyOjpyZWFkX3RzdihybmFzZXFfZmlsZSkKYGBgCgpgYGB7ciBwZWVrX3JuYXNlcSwgbGl2ZSA9IFRSVUV9CiMgV2hhdCBkb2VzIHRoZSBSTkEtc2VxIGRhdGEgZnJhbWUgbG9vayBsaWtlPwpybmFzZXFfZGZbMTo1LCAxOjVdCmBgYAoKRm9yIEdTVkEsIHdlIG5lZWQgYSBtYXRyaXguCgpgYGB7ciBybmFzZXFfbWF0LCBsaXZlID0gVFJVRX0Kcm5hc2VxX21hdCA8LSBybmFzZXFfZGYgJT4lCiAgdGliYmxlOjpjb2x1bW5fdG9fcm93bmFtZXMoImdlbmVfc3ltYm9sIikgJT4lCiAgYXMubWF0cml4KCkKYGBgCgoqTm90ZTogSWYgd2UgaGFkIGR1cGxpY2F0ZSBnZW5lIHN5bWJvbHMgaGVyZSwgd2UgY291bGRuJ3Qgc2V0IHRoZW0gYXMgcm93bmFtZXMuKgoKIyMgR1NWQQoKIVtdKGRpYWdyYW1zL2hhbnplbG1hbm5fZmlnMS5qcGcpCgoqKkZpZ3VyZSAxIGZyb20gW0jDpG56ZWxtYW5uIF9ldCBhbC5fICgyMDEzKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpLioqCgpZb3UgbWF5IG5vdGljZSB0aGF0IEdTVkEgaGFzIHNvbWUgY29tbW9uYWxpdGllcyB3aXRoIEdTRUEuClJhdGhlciB0aGFuIHJhbmtpbmcgZ2VuZXMgYmFzZWQgb24gc29tZSBzdGF0aXN0aWMgX3dlXyBzZWxlY3RlZCBhaGVhZCBvZiB0aW1lLCBHU1ZBIGZpdHMgYSBtb2RlbCBhbmQgcmFua3MgZ2VuZXMgYmFzZWQgb24gdGhlaXIgZXhwcmVzc2lvbiBsZXZlbCByZWxhdGl2ZSB0byB0aGUgc2FtcGxlIGRpc3RyaWJ1dGlvbi4KVGhpcyBpcyBhIHdheSBvZiBhc2tpbmcgaWYgYSBnZW5lIF9pXyBpcyBoaWdobHkgb3IgbG93bHkgZXhwcmVzc2VkIGluIGEgc2FtcGxlIF9qXyBpbiB0aGUgY29udGV4dCBvZiB0aGlzIGV4cGVyaW1lbnQgYW5kIHJhbmtpbmcgYWNjb3JkaW5nbHkgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpLgpUaGUgcGF0aHdheS1sZXZlbCBzY29yZSBjYWxjdWxhdGVkIGlzIGEgd2F5IG9mIGFza2luZyBob3cgZ2VuZXMgX3dpdGhpbl8gYSBnZW5lIHNldCB2YXJ5IGFzIGNvbXBhcmVkIHRvIGdlbmVzIHRoYXQgYXJlIF9vdXRzaWRlXyBvZiB0aGF0IGdlbmUgc2V0IChbTWFsaG90cmEuIDIwMThdKGh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS9kZWNvZGluZy1nZW5lLXNldC12YXJpYXRpb24tYW5hbHlzaXMtODE5M2EwY2ZkYTMpKS4gCihUaGlzIGlzIHNvbWV0aW1lcyBjYWxsZWQgYSBjb21wZXRpdGl2ZSB0ZXN0IGluIGdlbmUgc2V0IGVucmljaG1lbnQgbGl0ZXJhdHVyZS4pClRoZSBpbnR1aXRpb24gaGVyZSBpcyB0aGF0IHdlIHdpbGwgZ2V0IHBhdGh3YXktbGV2ZWwgc2NvcmVzIGZvciBlYWNoIHNhbXBsZSB0aGF0IGluZGljYXRlIGlmIGdlbmVzIGluIGEgcGF0aHdheSB2YXJ5IGNvbmNvcmRhbnRseSBpbiBvbmUgZGlyZWN0aW9uIChvdmVyZXhwcmVzc2VkIG9yIHVuZGVyZXhwcmVzc2VkIHJlbGF0aXZlIHRvIHRoZSBvdmVyYWxsIHBvcHVsYXRpb24pIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKS4KClRoZSBvdXRwdXQgaXMgYSBnZW5lIHNldCBieSBzYW1wbGUgbWF0cml4IG9mIEdTVkEgc2NvcmVzLgoKIyMjIFBlcmZvcm0gR1NWQQoKYGBge3IgcnVuX2dzdmF9CmdzdmFfcmVzdWx0cyA8LSBnc3ZhKHJuYXNlcV9tYXQsIAogICAgICAgICAgICAgICAgICAgICBoYWxsbWFya3NfbGlzdCwKICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdzdmEiLAogICAgICAgICAgICAgICAgICAgICAjIEFwcHJvcHJpYXRlIGZvciBvdXIgdHJhbnNmb3JtZWQgZGF0YSBvbiBsb2cyLWxpa2Ugc2NhbGUKICAgICAgICAgICAgICAgICAgICAga2NkZiA9ICJHYXVzc2lhbiIsICAKICAgICAgICAgICAgICAgICAgICAgIyBNaW5pbXVtIGdlbmUgc2V0IHNpemUKICAgICAgICAgICAgICAgICAgICAgbWluLnN6ID0gMTUsIAogICAgICAgICAgICAgICAgICAgICAjIE1heGltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtYXguc3ogPSA1MDAsCiAgICAgICAgICAgICAgICAgICAgICMgQ29tcHV0ZSBHYXVzc2lhbi1kaXN0cmlidXRlZCBzY29yZXMKICAgICAgICAgICAgICAgICAgICAgbXguZGlmZiA9IFRSVUUpCmBgYAoKKipOb3RlOiB0aGUgYGdzdmEoKWAgZG9jdW1lbnRhdGlvbiBzYXlzIHdlIGNhbiB1c2UgYGtjZGYgPSAiR2F1c3NpYW4iYCBpZiB3ZSBoYWQgUk5BLXNlcSBsb2ctQ1BNcywgbG9nLVJQS01zIG9yIGxvZy1UUE1zLCBidXQgd2Ugd291bGQgdXNlIGBrY2RmID0gIlBvaXNzb24iYCBvbiBpbnRlZ2VyIGNvdW50cy4qKgoKYGBge3IgZ3N2YV9wZWVrfQojIExldCdzIGV4cGxvcmUgd2hhdCB0aGUgb3V0cHV0IG9mIGdzdmEoKSBsb29rcyBsaWtlCmdzdmFfcmVzdWx0c1sxOjUsIDE6NV0KYGBgCgojIyMgQSBub3RlIG9uIGdlbmUgc2V0IHNpemUKIApPZnRlbiB0aGUgc2NvcmVzIG9mIGdlbmUgc2V0IGVucmljaG1lbnQgbWV0aG9kcyBhcmUgbm90IGNvbXBhcmFibGUgYmV0d2VlbiBnZW5lIHNldHMgb2YgZGlmZmVyZW50IHNpemVzLgpMZXQncyBkbyBhbiBleHBlcmltZW50IHVzaW5nIHJhbmRvbWx5IGdlbmVyYXRlZCBnZW5lIHNldHMgdG8gZXhwbG9yZSB0aGlzIGlkZWEgYSBiaXQgbW9yZS4KCldlIG5lZWQgdG8gZ2V0IGEgY29sbGVjdGlvbiBvZiBhbGwgcG9zc2libGUgZ2VuZXMgd2Ugd2lsbCBzYW1wbGUgZnJvbSB0byBjcmVhdGUgcmFuZG9tIGdlbmUgc2V0cy4KQmVjYXVzZSB3ZSdyZSBkb2luZyBzb21lIHJhbmRvbSBzYW1wbGluZywgd2UgbmVlZCB0byBzZXQgYSBzZWVkIGZvciB0aGlzIHRvIGJlIHJlcHJvZHVjaWJsZS4KCmBgYHtyIHJhbmRvbV9zZXR1cH0KIyBVc2UgYWxsIHRoZSBnZW5lIHN5bWJvbHMgaW4gdGhlIGRhdGFzZXQgYXMgdGhlIHBvb2wgb2YgcG9zc2libGUgZ2VuZXMKYWxsX2dlbmVzIDwtIHJvd25hbWVzKHJuYXNlcV9tYXQpCgojIFNldCBhIHNlZWQgZm9yIHJlcHJvZHVjaWJpbGl0eQpzZXQuc2VlZCgyMDIwKQpgYGAKCk91ciBtaW5pbXVtIGdlbmUgc2V0IHNpemUgZWFybGllciB3YXMgMTUgZ2VuZXMgYW5kIG91ciBtYXhpbXVtIGdlbmUgc2V0IHNpemUgd2FzIDUwMCBnZW5lcy4gCldlJ2xsIHVzZSB0aGUgc2FtZSBtaW5pbXVtIGFuZCBtYXhpbXVtIHZhbHVlcyBmb3Igb3VyIHJhbmRvbSBnZW5lIHNldHMgYW5kIHNvbWUgdmFsdWVzIGluIGJldHdlZW4uCgpgYGB7ciBnZW5lX3NldF9zaXplc30KIyBNYWtlIGEgbGlzdCBvZiBpbnRlZ2VycyB0aGF0IGluZGljYXRlIHRoZSByYW5kb20gZ2VuZSBzZXQgc2l6ZXMKZ2VuZV9zZXRfc2l6ZSA8LSBsaXN0KDE1LCAyNSwgNTAsIDEwMCwgMjUwLCA1MDApCmBgYAoKRm9yIGVhY2ggZ2VuZSBzZXQgc2l6ZSwgd2Ugd2lsbCBnZW5lcmF0ZSAxMDAgcmFuZG9tIGdlbmUgc2V0cy4KCmBgYHtyIHJhbmRvbV9nZW5lX3NldHN9CiMgU2V0IG51bWJlciBvZiByZXBsaWNhdGVzCm5yZXBzIDwtIDEwMAojIEdlbmVyYXRlIDEwMCByYW5kb20gZ2VuZSBzZXRzIG9mIGVhY2ggc2l6ZQpyYW5kb21fZ2VuZV9zZXRzIDwtIHJlcChnZW5lX3NldF9zaXplLCBucmVwcykgJT4lICAjIFJlcGVhdCBnZW5lIHNpemVzIHNvIHdlIHJ1biBgbnJlcHNgIHRpbWVzIAogIHB1cnJyOjptYXAoCiAgICAjIFNhbXBsZSB0aGUgdmVjdG9yIG9mIGFsbCBnZW5lcywgY2hvb3NpbmcgdGhlIG51bWJlciBvZiBpdGVtcyBzcGVjaWZpZWQKICAgICMgaW4gdGhlIGVsZW1lbnQgb2YgZ2VuZSBzZXQgc2l6ZQogICAgfiBiYXNlOjpzYW1wbGUoeCA9IGFsbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIHNpemUgPSAueCkKICApCmBgYAoKVGhlIEhhbGxtYXJrcyBsaXN0IHdlIHVzZWQgZWFybGllciBzdG9yZWQgdGhlIGdlbmUgc2V0IG5hbWVzIGFzIHRoZSBuYW1lIG9mIHRoZSBsaXN0LCBzbyBsZXQncyBhZGQgbmFtZXMgdG8gb3VyIHJhbmRvbSBnZW5lIHNldHMgdGhhdCBpbmRpY2F0ZSB3aGF0IHNpemUgdGhleSBhcmUgYW5kIHNvIGBnc3ZhKClgIGRvZXNuJ3QgZ2V0IHVwc2V0LgoKYGBge3IgbmFtZV9yYW5kb21fZ2VuZV9zZXRzfQojIFdlIHdpbGwgaW5jbHVkZSB0aGUgc2l6ZSBvZiB0aGUgZ2VuZSBzZXQgaW4gdGhlIGdlbmUgc2V0IG5hbWUgCiMgU3RhcnQgYnkgdGFraW5nIHRoZSBsZW5ndGggb2YgZWFjaCBwYXRod2F5IGFuZCBhcHBlbmRpbmcgInBhdGh3YXlfIiB0byB0aGF0CiMgbnVtYmVyCmxlbmd0aHNfdmVjdG9yIDwtIHJhbmRvbV9nZW5lX3NldHMgJT4lCiAgIyBHZXQgdGhlIGxlbmd0aCBvZiBlYWNoIGdlbmUgc2V0IChudW1iZXIgb2YgZ2VuZXMpCiAgcHVycnI6Om1hcCh+IGxlbmd0aCgueCkpICU+JQogICMgTWFrZSBpdCAicGF0aHdheV88Z2VuZSBzZXQgc2l6ZT4iCiAgcHVycnI6Om1hcCh+IHBhc3RlMCgicGF0aHdheV8iLCAueCkpICU+JQogICMgUmV0dXJuIGEgdmVjdG9yCiAgcHVycnI6OmZsYXR0ZW5fY2hyKCkKCiMgQWRkIHRoZSBuYW1lcyBpbiBsZW5ndGhzX3ZlY3RvciB0byB0aGUgbGlzdCAtICJwYXRod2F5XzxnZW5lIHNldCBzaXplPiIKcmFuZG9tX2dlbmVfc2V0cyA8LSByYW5kb21fZ2VuZV9zZXRzICU+JSAKICAjIG1ha2UubmFtZXMoKSBhcHBlbmRzIGEgInZlcnNpb24iIGlmIHNvbWV0aGluZyBpcyBub3QgdW5pcXVlCiAgcHVycnI6OnNldF9uYW1lcyhubSA9IG1ha2UubmFtZXMobGVuZ3Roc192ZWN0b3IsIHVuaXF1ZSA9IFRSVUUpKQpgYGAKIApSdW4gR1NWQSBvbiBvdXIgZGF0YXNldCB3aXRoIHRoZSBzYW1lIHBhcmFtZXRlcnMgYXMgYmVmb3JlLCBidXQgbm93IHdpdGggcmFuZG9tIGdlbmUgc2V0cy4KCmBgYHtyIHJhbmRvbV9nc3ZhLCBsaXZlID0gVFJVRX0KcmFuZG9tX2dzdmFfcmVzdWx0cyA8LSBnc3ZhKHJuYXNlcV9tYXQsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgcmFuZG9tX2dlbmVfc2V0cywKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJnc3ZhIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgQXBwcm9wcmlhdGUgZm9yIG91ciB0cmFuc2Zvcm1lZCBkYXRhIG9uIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBsb2cyLWxpa2Ugc2NhbGUKICAgICAgICAgICAgICAgICAgICAgICAgICAgIGtjZGYgPSAiR2F1c3NpYW4iLCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIE1pbmltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWluLnN6ID0gMTUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBNYXhpbXVtIGdlbmUgc2V0IHNpemUKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1heC5zeiA9IDUwMCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgQ29tcHV0ZSBHYXVzc2lhbi1kaXN0cmlidXRlZCBzY29yZXMKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG14LmRpZmYgPSBUUlVFKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAKYGBgCgpOb3cgbGV0J3MgbWFrZSBhIHBsb3QgdG8gbG9vayBhdCB0aGUgZGlzdHJpYnV0aW9uIG9mIHNjb3JlcyBmcm9tIHJhbmRvbSBnZW5lIHNldHMuCkZpcnN0IHdlIG5lZWQgdG8gZ2V0IHRoaXMgZGF0YSBpbiBhbiBhcHByb3ByaWF0ZSBmb3JtYXQgZm9yIGBnZ3Bsb3QyYC4KCmBgYHtyIGxvbmdlcl9yYW5kb21fZ3N2YX0KIyBUaGUgcmFuZG9tIHJlc3VsdHMgYXJlIGEgbWF0cml4CnJhbmRvbV9sb25nX2RmIDwtIHJhbmRvbV9nc3ZhX3Jlc3VsdHMgJT4lCiAgZGF0YS5mcmFtZSgpICU+JQogICMgR2VuZSBzZXQgbmFtZXMgYXJlIHJvd25hbWVzCiAgdGliYmxlOjpyb3duYW1lc190b19jb2x1bW4oImdlbmVfc2V0IikgJT4lCiAgIyBHZXQgaW50byBsb25nIGZvcm1hdAogIHRpZHlyOjpwaXZvdF9sb25nZXIoY29scyA9IC1nZW5lX3NldCwgCiAgICAgICAgICAgICAgICAgICAgICBuYW1lc190byA9ICJLaWRzX0ZpcnN0X0Jpb3NwZWNpbWVuX0lEIiwKICAgICAgICAgICAgICAgICAgICAgIHZhbHVlc190byA9ICJnc3ZhX3Njb3JlIikgJT4lCiAgIyBSZW1vdmUgdGhlIC52ZXJzaW9uIGFkZGVkIGJ5IG1ha2UubmFtZXMoKQogIGRwbHlyOjptdXRhdGUoZ2VuZV9zZXQgPSBzdHJpbmdyOjpzdHJfcmVtb3ZlKGdlbmVfc2V0LCAiXFwuLioiKSkgJT4lCiAgIyBBZGQgYSBjb2x1bW4gdGhhdCBrZWVwcyB0cmFjayBvZiB0aGUgZ2VuZSBzZXQgc2l6ZQogIGRwbHlyOjptdXRhdGUoZ2VuZV9zZXRfc2l6ZSA9IHN0cmluZ3I6OndvcmQoZ2VuZV9zZXQsIDIsIHNlcCA9ICJfIikpICU+JQogICMgV2Ugd2FudCB0byBwbG90IHNtYWxsZXN0IG5vLiBnZW5lcyAtPiBsYXJnZXN0IG5vLiBnZW5lcwogIGRwbHlyOjptdXRhdGUoZ2VuZV9zZXRfc2l6ZSA9IGZhY3RvcihnZW5lX3NldF9zaXplLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGV2ZWxzID0gYygxNSwgMjUsIDUwLCAxMDAsIDI1MCwgNTAwKSkpCgpgYGAKCkxldCdzIG1ha2UgYSB2aW9saW4gcGxvdCBzbyB3ZSBjYW4gbG9vayBhdCB0aGUgZGlzdHJpYnV0aW9uIG9mIHNjb3JlcyBieSBnZW5lIHNldCBzaXplLgoKYGBge3IgcmFuZG9tX3Zpb2xpbn0KIyBWaW9saW4gcGxvdCBjb21wYXJpbmcgR1NWQSBzY29yZXMgb2YgZGlmZmVyZW50IHJhbmRvbSBnZW5lIHNldCBzaXplcwpyYW5kb21fbG9uZ19kZiAlPiUKICBnZ3Bsb3QyOjpnZ3Bsb3QoZ2dwbG90Mjo6YWVzKHggPSBnZW5lX3NldF9zaXplLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHkgPSBnc3ZhX3Njb3JlKSkgKwogICMgTWFrZSBhIHZpb2xpbiBwbG90IHRoYXQncyBhIHByZXR0eSBibHVlIQogIGdncGxvdDI6Omdlb21fdmlvbGluKGZpbGwgPSAiIzk5Q0NGRiIsIGFscGhhID0gMC41KSArCiAgIyBBZGQgYSBwb2ludCB3aXRoIHRoZSBtZWFuIHZhbHVlCiAgZ2dwbG90Mjo6c3RhdF9zdW1tYXJ5KAogICAgZ2VvbSA9ICJwb2ludCIsCiAgICBmdW4gPSAibWVhbiIsCiAgICAjIENoYW5nZSB0aGUgYWVzdGhldGljcyBvZiB0aGUgcG9pbnRzCiAgICBzaXplID0gMywKICAgIGNvbG9yID0gIiMwMDY2Q0MiLAogICAgc2hhcGUgPSAxOAogICkgKwogICMgRmxpcCB0aGUgYXhlcwogIGdncGxvdDI6OmNvb3JkX2ZsaXAoKSArCiAgZ2dwbG90Mjo6bGFicyh0aXRsZSA9ICJSYW5kb20gR2VuZSBTZXQgR1NWQSBTY29yZXMiLAogICAgICAgICAgICAgICAgeCA9ICJnZW5lIHNldCBzaXplIiwKICAgICAgICAgICAgICAgIHkgPSAiR1NWQSBzY29yZSIpICsKICBnZ3Bsb3QyOjp0aGVtZV9idygpCmBgYAoKV2hhdCBkbyB5b3Ugbm90aWNlIGFib3V0IHRoZXNlIGRpc3RyaWJ1dGlvbnM/CkhvdyBtaWdodCB5b3UgdXNlIHRoaXMgaW5mb3JtYXRpb24gdG8gaW5mb3JtIHlvdXIgaW50ZXJwcmV0YXRpb24gb2YgR1NWQSBzY29yZXM/CgojIyMgSG93IGNhbiB5b3UgdXNlIHRoZXNlIHNjb3Jlcz8KCklmIHlvdSBkaWQgaGF2ZSBncm91cHMgaW5mb3JtYXRpb24gZm9yIHlvdXIgc2FtcGxlcywgeW91IGNvdWxkIHRlc3QgZm9yIHBhdGh3YXkgc2NvcmVzIGRpZmZlcmVuY2VzIGJldHdlZW4gZ3JvdXBzLgpIZXJlJ3MgW2FuIGV4YW1wbGVdKGh0dHBzOi8vaHRtbHByZXZpZXcuZ2l0aHViLmlvLz9odHRwczovL2dpdGh1Yi5jb20vQWxleHNMZW1vbmFkZS9PcGVuUEJUQS1hbmFseXNpcy9ibG9iLzliNDRiZjFjMTg2YjMxMjZiMTZkYmU1Yjg3NzU2YjNlYWUzZmVlYzIvYW5hbHlzZXMvZ2VuZS1zZXQtZW5yaWNobWVudC1hbmFseXNpcy8wMi1tb2RlbC1nc2VhLm5iLmh0bWwpIG9mIHRoYXQgZnJvbSB0aGUgT3BlblBCVEEgcHJvamVjdCBpdHNlbGYhIAoKWW91IGNhbiBhbHNvIHZpc3VhbGl6ZSB0aGlzIG1hdHJpeCBpbiBhIGhlYXRtYXAuCkhlcmUncyBhIGZpZ3VyZSBmcm9tIHRoZSBPcGVuUEJUQSBwcm9qZWN0LCB3aGVyZSB0aGUgbWlkZGxlIHBhbmVsIGlzIGEgaGVhdG1hcCBvZiBHU1ZBIHNjb3JlcyB0aGF0IHdlcmUgc2lnbmlmaWNhbnRseSBkaWZmZXJlbnQgYmV0d2VlbiBoaXN0b2xvZ2llcy4KCiFbXShodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vQWxleHNMZW1vbmFkZS9PcGVuUEJUQS1hbmFseXNpcy85YjQ0YmYxYzE4NmIzMTI2YjE2ZGJlNWI4Nzc1NmIzZWFlM2ZlZWMyL2ZpZ3VyZXMvcG5ncy90cmFuc2NyaXB0b21pYy1vdmVydmlldy5wbmcpCgpbKkNvZGUgaGVyZS4qXShodHRwczovL2dpdGh1Yi5jb20vQWxleHNMZW1vbmFkZS9PcGVuUEJUQS1hbmFseXNpcy9ibG9iLzliNDRiZjFjMTg2YjMxMjZiMTZkYmU1Yjg3NzU2YjNlYWUzZmVlYzIvZmlndXJlcy9zY3JpcHRzL3RyYW5zY3JpcHRvbWljLW92ZXJ2aWV3LlIjTDEwNikKCiMjIyBXcml0ZSByZXN1bHRzIHRvIGZpbGUKCmBgYHtyIHdyaXRlX2dzdmFfcmVzdWx0c30KZ3N2YV9yZXN1bHRzICU+JQogIGFzLmRhdGEuZnJhbWUoKSAlPiUKICB0aWJibGU6OnJvd25hbWVzX3RvX2NvbHVtbigicGF0aHdheSIpICU+JQogIHJlYWRyOjp3cml0ZV90c3YoZmlsZSA9IGdzdmFfcmVzdWx0c19maWxlKQpgYGAKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=